Method and system for constructing an internet-based imaging system

ABSTRACT

A system and method is presented that uses software logic (executing in one or more servers) to inter-connect mobile devices and location-deriving signal generating devices to collectively act as an Imaging System that outputs a periodic succession of images containing information about the preferences and purchase and transaction history of consumers. The Imaging System so constructed is called a SoftCamera in a fashion analogous to a conventional light-based camera. Images output by the SoftCamera may be transmitted to third-party providers or used to send information and recommendations to consumers. The SoftCamera thus may be used as a Recommender System in certain embodiments.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a non-provisional and claims priority to U.S.Provisional No. 62/023,457, filed Jul. 11, 2014 entitled “A System andMethod for Creating and Rendering Mediated Reality Representations ofNetworked Spaces Using a SoftCamera” and is also a non-provisional andclaims priority to U.S. Provisional Application No. 62/113,605, filedFeb. 19, 2015 entitled “Mediated Representations of Real and VirtualSpaces”, the entirety of each prior application being incorporated byreference herein.

This application is also related to the following:

1. U.S. application Ser. No. 14/701,874 (Our Ref.: 12000/5) entitled “ASystem and Method for Mediating Representations with Respect toPreferences Of a Party Not Located in the Environment” filed May 1,2015.2. U.S. application Ser. No. 14/701,858 (Our Ref.: 12000/2) entitled “ASystem and Method for Mediating Representations with Respect to UserPreferences” filed May 1, 2015.3. U.S. patent application Ser. No. 14/701,883 (Our Ref.: 12000/6)entitled “A System and Method for Inferring the Intent of a User WhileReceiving Signals On a Mobile Communication Device From a BroadcastingDevice” filed May 1, 2015.

The entirety of each prior application is incorporated by referenceherein.

BACKGROUND

As indoor location-sensing systems become more prevalent, theopportunity arises of using mobile devices to sense various geographicalaspects of an area, derive proximity and location information about thecontents within said area, and use such information to provide customerswith a personalized and customized experience.

With respect to location-deriving signal generating devices andtechnologies, we mention Indoor GPS systems and Bluetooth-Low Energy(BLE) devices (e.g., beacons) as examples that allow more accuratelocation sensing than conventional GPS systems.

A BLE device typically broadcasts messages at a periodic rate within acoverage area with a radius of a few meters. Mobile devices within saidcoverage may then receive these broadcasts and applications (“apps”)within the mobile device are made aware of the broadcasts by said mobiledevice's operating system. The apps then process the incoming broadcastmessages and convey them in turn to servers in a wide-area networkwherein further processing of the signals takes place.

SUMMARY

In accordance with one aspect of the invention a system and method isprovided to inter-connect mobile devices and location-deriving signalgenerating devices into a specific arrangement to yield an ImagingSystem.

In accordance with one aspect of the invention the inter-connectedcomponents of the Imaging System are geographically distributed using aprivate data network such as the Internet.

In accordance with one aspect of the invention the inter-connectionlogic that constructs the Imaging System resides in a collection ofservers connected to a wide area network (also known as a CloudComputing System).

In accordance with one aspect of the invention the Imaging Systemproduces datasets that are called “images”.

Furthermore, the Imaging System may produce a succession of images at apre-determined and configurable rate.

In accordance with one aspect of the invention a system and method isprovided whereby information received from one or more mobile deviceslocated within a demarcated geographical area is aggregated to producesequences of images of said geographical area.

In accordance with one aspect of the invention the location-derivinginformation received from mobile devices originates from Bluetooth-LowEnergy (BLE) devices, also known as beacons, e.g., Gimbals, Estimote;furthermore also from devices based on the industry-standardspecification called iBeacon.

In accordance with one aspect of the invention the mobile devicestransmitting the received information are mobile communication devicessuch as smartphones, tablet computers, Personal Digital Assistants(PDA), smart watch, smart glasses, mobile terminals, mobile computersand mobile phones.

In accordance with one aspect of the invention the aggregation ofinformation further comprises integrating information received fromonline data feeds, said data feeds emanating from websites, private datasources or data repositories, news feeds, and public and privatedatabases.

In accordance with one aspect of the invention aggregation ofinformation comprises the identification of different areas of thegeographical area distinguished by the number of mobile devices in saidareas over a pre-determined or configurable time period.

In accordance with one aspect of the invention the aggregation ofinformation further comprises integration of information received fromdescriptions of the geographical area, possibly including inventory andproduct or item information.

In accordance with one aspect of the invention the aggregation ofinformation further comprises information obtained by analysis of onlinedata feeds.

In accordance with one aspect of the invention the analysis of onlinedata feeds comprises use of machine learning procedures and techniques.

In accordance with one aspect of the invention an image is a descriptionof the layout of the geographical space, e.g., location of mobiledevices in a geographical area and the location of BLE devices, theitems in the space, shelves, aisles, etc.

In accordance with one aspect of the invention an image is producedperiodically with a rate determined by system policy, administratorpolicy or a policy governed by the cost and limitations of computing andnetworking machinery.

In accordance with one aspect of the invention the temporal sequence ofimages of a geographical area are stored in a historical datarepository.

In accordance with one aspect of the invention said historical datarepository is used to predict future areas of the geographical area thatwill be distinguished by the number of mobile devices.

In accordance with one aspect of the invention said historical datarepository is used to predict the future sale of items in thegeographical area.

In accordance with one aspect of the invention the predictions ofregions of the geographical area distinguished by number of mobiledevices is used to manage the merchandizing and inventory of itemscontained in the geographical area.

In accordance with one aspect of the invention the location of a mobiledevice as indicated by an image is used determine its proximity to oneor more, i.e., a collection of, items.

In accordance with one aspect of the invention the proximity of a mobiledevice to a collection of items is used to isolate one or more specificitems, designated as “items of interest”.

In accordance with one aspect of the invention mobile devices present inan image are uniquely associated with one or more identifiers (“useridentifier list”) available from the mobile devices, or by matching saidavailable identifiers with data from historical, loyalty, or CustomerRelationship Management (CRM) datasets.

In accordance with one aspect of the invention the user identifier listis used to identify consumers, i.e., one or more identifiers signifyingan electronic address, e.g., email address, Twitter handle, Skypeaddress, etc.

In accordance with one aspect of the invention the identified consumersare associated with the identified items of interest.

In accordance with one aspect of the invention the identified consumersare sent electronic messages related to the items of interest.

In accordance with one aspect of the invention the electronic messagesmay be mobile push messages of various types such as full, interactive,in-app push, rich push, etc.), email messages, notifications, or textmessages.

In accordance with one aspect of the invention the electronic messagesmay be delivered while the device is within the demarcated geographicalarea or after the device has exited the geographical area.

In accordance with one aspect of the invention the items of interestlist is augmented with additional items based on analysis of online datafeeds and/or derived from machine-learning techniques.

In accordance with one aspect of the invention the electronic messagesrelate to the augmented list of items of interest.

In accordance with one aspect of the invention the geographical area isdemarcated by one or more geo-fences whose boundaries are determinedwith respect to the range of the BLE devices within the geographicalarea (“BLE-geo-fence”).

In accordance with one aspect of the invention a mobile device within aBLE-geo-fence is identified as a “marked device”.

In accordance with one aspect of the invention the marked devicetriggers special software logic (“orchestration script”) to bedownloaded to the mobile device from one or more servers (“servercomplex”) connected to the mobile device by a wide area network.

In accordance with one aspect of the invention the orchestration scriptis pre-loaded to the mobile device preceding its time of entry to theBLE-geo-fence.

In accordance with one aspect of the invention the orchestration scriptis launched when the mobile device encounters a first BLE device, or ata pre-determined or pre-configured time within the BLE-geo-fence.

In accordance with one aspect of the invention the orchestration scriptis launched when the mobile device recognizes the loss of connectivityto the cloud.

In accordance with one aspect of the invention the exit of a mobiledevice from within a BLE-geo-fence triggers special software logic(“post-visit orchestration script”) to be executed in the servercomplex.

In accordance with one aspect of the invention the post-visitorchestration script causes electronic messages to be delivered to themobile device or to the identified consumers.

In accordance with one aspect of the invention the image derived for ademarcated geographical area is rendered on laptops, personal computers,tablets, and mobile devices in a visual representation for human users.

In accordance with one aspect of the invention the image is delivered toexternal multiple computers and/or mobile devices.

In accordance with one aspect of the invention the delivery of an imageto other devices may be contemporaneous with the location of one or moredevices within the demarcated geographical area.

In accordance with one aspect of the invention an image delivered toother devices may additionally contain recommendations, instructions orinformational objects inserted manually, i.e., curated by a human user,or by a policy curated by a human, or by a policy derived automaticallyby the system based on historical analysis of past manual insertions.

In accordance with one aspect of the invention the image derived for ademarcated geographical area is made available to third-party serviceproviders in a machine-readable format.

In accordance with one aspect of the invention the items of interestlist is rank-ordered, said ranking determined by utilizing informationfrom online data analysis or machine-learning techniques.

In accordance with an aspect of the invention the areas with thedemarcated geographical area that are distinguished by number of mobiledevices are used to manage advertising displays in said areas.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example Planogram.

FIGS. 2, 3, 4 and 5 show a sequence of images produced by successiveenhancements. Each image is produced at one click of the Shutter SpeedClock of the SoftCamera (detailed later). In particular, FIG. 2 shows animage obtained by enhancing the Planogram shown in FIG. 1.

FIG. 3 shows an enhancement of the image of FIG. 2 by addition of “hotzones”.

FIG. 4 shows an enhancement of the image of FIG. 3 by zooming in on aparticular mobile device of consumer “John”.

FIG. 5 shows an enhancement by further zooming in on the preferences forconsumer “John”.

FIG. 6 shows an example of a Planogram for a given DGA.

FIG. 7 shows an image obtained from a Planogram by adding moreinformation (i.e., mobile devices).

FIG. 8 shows a conventional view of a light and photographic film-basedcamera.

FIG. 9 shows the components of the SoftCamera in analogy to aconventional camera.

FIG. 10 shows a general architecture of the SoftCamera.

FIG. 11 shows the internal modules of the SoftCamera.

FIG. 12 shows an example of a Script Table for an Orchestration Script.

FIG. 13 illustrates various components of an illustrativecomputing-based device in which embodiments of various servers and/orclients as described herein may be implemented.

DETAILED DESCRIPTION Motivation

Consider a retail establishment that displays its wares in shelvesarranged by aisles. Customers browse the items being displayed on theshelves. A store manager watches the consumers and their actions on adisplay device connected to a camera installed in the store. Perhaps heis situated in a remote location from the store with said display deviceconnected to the camera by private data networks such as the Internet.

Such an arrangement allows a store manager to oversee store operations,e.g., how the sales people are interacting with the customers, andsecurity, e.g., the items on the shelves are not being stolen.

There are, however, other far-reaching benefits. Over time the storemanager notices which aisles see more traffic, where customers usuallycongregate and what items garner more attention. He gets to recognizerepeat customers, items frequently bought in conjunction with otheritems, etc.

A thoughtful manager may assimilate such information and codify it intocertain decisions that reflect his said assimilation. He may re-arrangehis items in a different way, re-arrange his aisles and shelves, givedifferent instructions to his sales staff, issue real-time instructionsto his staff to cater to a customer while the customer is lingering byan item in the store, pro-actively offer other items to customersbrowsing in the store, entreat passersby to enter the store with anenticing offer, said offer relevant and pertaining to the interests ofthe browsing customer or passerby.

The manager's decisions are rooted in the information that he gathersfrom the camera, the information he obtains from his inventory, thearrangement and layout of his store, data about current and pastpurchases, and trends in user interests as gleaned from one or moreinformation sources accessible to the manager.

The present invention is concerned with inter-connectinglocation-deriving signal generating devices and mobile devices to createa system that produces “images” of geographical areas; said imagescontaining information about the customers in said area, theirinterests, purchase intent, movements, etc. Said images may be furtherenhanced by adding inferences made by the system based on variousdatasets such as CRM (Customer Relationship Management) data, pastpurchase data or Loyalty data. Said images may be manipulated bycomputer systems and/or provided to third-party providers for furtherprocessing.

Said inter-connection is achieved by software logic executing in one ormore computers, the inter-connected components being in connection withsaid software logic via private data networks such as the Internet.

The system comprising the software logic, the inter-connected mobiledevices and the location-deriving signal generating devices togetherform an Imaging System called the SoftCamera.

The SoftCamera produces images akin to a conventional light andphotographic-film based camera, said images comprising data that can bemanipulated by computers, e.g., to make recommendations or offers toconsumers.

The SoftCamera uses a special dataset in a crucial role. The dataset iscalled a Planogram. The data within the dataset relates to the layoutand items contained within a geographic area. Planograms nowadays findusage in retail store applications. However, their use need not belimited to the retail domain only. The data in a Planogram may bemanipulated by computer programs and also may be rendered on displaydevices for human consumption. Consider by way of example a Planogram ofa retail establishment rendered on a display device as shown in FIG. 1.The figure shows various aisles of the store and may list items on theshelves, etc. The figure also shows several location sensing devicesplaced within the store.

We may now enhance the data in a Planogram by adding more information toit. If we enhance the Planogram rendered in FIG. 1 and render it on adisplay device we may get a rendering as shown in FIG. 2. For ease ofterminology in forthcoming descriptions we will abbreviate by simplystating that FIG. 2 enhances FIG. 1. An enhanced Planogram is called animage.

One may successively enhance images. For example, FIG. 3 enhances theimage shown in FIG. 2 by integrating the positions of the mobile deviceswithin the store over a period of time and representing the resultsvisually (also called “hot zones”).

FIG. 4 further enhances the image of FIG. 3 by enhancing one mobiledevice and calculating information about the owner (“John”) of saidmobile device, his purchase history, and the advertisements he was shownrecently in his online browsing sessions (obtained from an externaldataset provided to the system).

FIG. 5 continues the enhancement of customer John and shows the resultof calculating John's items of interest based on his movements withinthe store (“linger time”) and his past purchases, etc.

It may thus be seen that a system that can generate a series of imagessuch as FIG. 5 above at a periodic rate may provide valuable informationthat may be used by marketing professionals, for example.

Introduction

The following detailed descriptions are made with respect to the variousfigures included in the application and may refer to specific examplesin said drawings; however, it is to be noted that such specific cases donot limit the generality of the various aspects of the invention.

Certain figures pertain to a specific implementation of the presentinvention. These figures are included in the application to show aparticular implementation of some of the embodiments of the presentinvention and not intended to limit the generality of the variousaspects of the present invention.

The present invention is partly motivated by the advent of devices withcomputational capabilities applied to physical areas. This trend seemsto have been started with cell towers installed for supporting mobilecommunications but also used for locating mobile devices in geographicalareas. Global Positioning Systems (GPS) further improved the accuracy oflocation identification. Smaller-sized cell tower technologies such aspico-cells, femto-cells, etc., have also been used for mobilecommunications and for location tracking in indoor spaces. Indoor GPSsystems are a further example of location sensing technologies. Wi-Firouters and access points have also been used for determining locationsof devices. Recently, so-called Bluetooth Low Energy (BLE) devicesprovide improved location tracking capabilities in indoor spaces.

The descriptions that follow use the language and terminology of BLEdevices; however, said descriptions are only for descriptive reasons andmay not be taken to limit the inventions claimed herein. In particular,we will refer to any device that broadcasts signals that may be receivedby mobile devices and which contain information from which the locationof the mobile device can be derived as a location-deriving signalgenerator. Thus, any reference herein to a BLE is presented by way ofillustration only and more generally may refer to any suitablelocation-deriving signal generators, including but not limited to theaforementioned cell towers, GPS satellites, Wi-Fi routers and otheraccess points, and so on.

As a mobile device roams, it goes out of range of certain signals andcomes into the range of other signals from location sensing devices.Thus, at any given time, an area of geography may be determined whereina mobile device is able to receive signals from one or more locationsensing devices. Such a coverage area may be demarcated by defining ageo-fence whose surface area subsumes the coverage area. We will referto such an arrangement wherein a geographical area that has locationsensing devices installed within it and has one or more geo-fences as aDemarcated Geographical Area (DGA).

As an example of a DGA consider a retail establishment that has multipleBLE devices, e.g., Gimbals, installed within its premises, and has ageo-fence defined as a boundary around its premises in such a way thatthe signals from one or more of the BLE devices reach all parts of theDGA.

It should also be noted that no assumption is being made as to where theone or more BLE devices are installed within a DGA. In particular, saiddevices may be installed anywhere as long as their broadcast signals maybe received within the DGA. It should also be noted that a DGA mightcover a wide geographical area and may contain several geo-fences,possibly intersecting, subsuming and overlapping. For example, we may beinterested in a DGA that covers homes of customers and their journeys toa store and to capture such a wide area one may need to use severalgeo-fences and BLE devices covering multiple areas.

Mobile devices receive the signals broadcast by BLE devices in a DGA.Generally, mobile devices support applications (“apps”), and said appsmay operate using data received from the signals transmitted by one ormore said devices. In some cases the app must first register to receivesignals from the one or more devices. When the mobile device's operatingsystem receives such signals, it makes the registered applications awareof the receipt of said signals. When these applications are made awareof a received location-deriving signal, they may—transmit the dataembodied in said location-deriving signal to one or more servers using anetwork connection to a wide area network, e.g., a cloud infrastructure,wherein the servers assemble the received data into one or moredatasets.

FIG. 6 shows an example of a Planogram of a retail store. The oval 100represents a geo-fence that has been defined around the physicalenvironment 200. Thus, the physical environment may be thought of as aDGA. The environment contains aisles marked 500, 600 and 700. The aisleshave shelves containing items enumerated as I11, I21, etc. The entry tothe environment is marked as 300. Finally, BLE devices are installed andshown as 1000, 1001, 1002, 1003, and 1004. Thus, there is a BLE devicefor the exit, one by the cashier's station and one BLE device each forthe aisles. There is no significance attached to the placement of theBLE devices in this invention; it is assumed that management makes adetermination to install the BLE devices at particular locations withinthe DGA.

It is a characteristic of BLE devices that their transmitted signalscontain a signal strength indicator (called RSS denoting Received SignalStrength). It is known from prior art that a mobile device (or a serverin connection with said mobile device) may use the RSS signal from a BLEdevice to calculate the distance and location of said mobile device fromsaid BLE device.

A mobile device may receive RSS indicia from several BLE devices butsaid mobile device may be programmed to make a determination as to whichBLE device is the closest and chose a BLE device on the basis of such adetermination.

It is assumed that a Planogram such as FIG. 6 is provided to the systembeing described, i.e., it is an input to the system.

FIG. 7 shows the Planogram of FIG. 6 but it has now been enhanced by theaddition of mobile devices marked as 2000, 2001, 2002, 2003, 2004 and2005. A Planogram that has been enhanced with additional informationwill be referred to as an “image”.

One important aspect of images produced by the system described hereinis that they can be generated to contain information that can beautomatically processed, i.e., processed by other computer programswithout human intervention. For example, one may generate images of theretail store in such a manner that a congregation of customers may bedetected at a certain location in the store by a computer program thatanalyzes said images, and an alert may be generated automatically to apre-designated terminal or device, e.g., a manager's station. Similarly,a particular user moving in the store (as indicated by successivelocations of his mobile device) may be captured in an image that alsolists his preferences, i.e., items he is lingering by and may be ofinterest to him. Finally, it is to be noted that images may be generatedat periodic intervals to capture “snapshots” of the DGA over time, theperiod of such intervals being determined by administrator or systempolicy. This notion of periodic images is discussed in more detaillater.

SoftCamera

A video camera produces images (successive frames) of lighted objectsthat may be used to manage items, recognize people, etc. The merging ofthe physical and digital worlds, as entailed by the advent of BLEdevices, has provided the opportunity for creating images that capturecustomer-specific information.

The present invention is concerned with inter-connectinglocation-deriving signal generating devices and mobile devices to createa system that produces “images” of geographical areas; said imagescontaining information about the customers in said area, theirinterests, purchase intent, movements, etc. Said images may bemanipulated by computer systems and/or provided to third-party providersfor further processing.

Said inter-connection is achieved by software logic executing in one ormore computers, the inter-connected components being in connection withsaid software logic via any combination of wireless and wiredcommunication networks including, for instance, private data networkssuch as the Internet.

The system comprising the software logic, the inter-connected mobiledevices and the location-deriving signal generating devices togetherform an Imaging System called the SoftCamera.

The SoftCamera produces images akin to a conventional light andphotographic-film based camera, said images comprising data that can bemanipulated by computers, e.g., to make recommendations or offers toconsumers.

One embodiment of the present invention utilizes location sensingdevices of the type BLE and mobile devices of the type smartphones(e.g., iPhone, Android phone, etc.), to construct the SoftCamera.

FIG. 8 shows a conventional camera. Light from a light source 100reflects off an object 200 into a lens 300 that collects and providesthe incident light to a camera body 400. The camera body may usespecially processed film, e.g., color film as opposed to black and whilefilm, to enhance certain properties of the incident light. The lightrays then impinge on a photographic film 600 wherein the image 700 isproduced.

FIG. 9 shows the components of one illustrative example of theSoftCamera which is analogous to a conventional camera. Location sensingdevices such as the BLE serve analogously to light sources inconventional cameras in the sense that their transmitted messages “lightup” the receiver and operating system (OS) of the mobile device, whichmay be thought of as the “object”. The transmitter of the mobile devicethen may be thought of as the “lens” that provides signals to the server400 (“camera body”), generally over more wireless networks that mayinclude, for example, a cellular network. The camera body in order toenhance the received signals from the “lens” may use various datasets500. Such datasets may then be said to perform the role of speciallypre-processed film in a conventional camera. The input Planogram 600could be taken as the analog of photographic film and the enhancedinformation produced after processing may be taken as analogous to theimages 700 of the SoftCamera.

The datasets 500 may be pre-provisioned in the server 400 or they may beotherwise communicated to the server 400, possibly though notnecessarily, from the DGA itself over one or more communication systems.In some embodiments one or more of the datasets 500 may be communicatedto the server 400 in real-time, e.g., at the same time the mobiledevices are providing signals to the server 400.

It is to be noted that as mobile devices enter and exit a DGA, theamount of information entering the body of the SoftCamera may vary. Thismay be referred to as the focusing power of the lens varying over time(just as a light camera lens may lose focusing power due to changes inambient light).

The SoftCamera produces an image 700 for a DGA containing one or moremobile devices. Since one may be interested in multiple DGAs distributedover a wide geographical area (e.g., in a city wide service) theSoftCamera may produce multiple images in parallel for multiple DGAs.

The images 700 so generated contain computer-generated data objects.When computer programs process and manipulate the images of a DGA andrender the results on display devices, we refer to such processes asrendering an image—much like images captured by (visible) light camerasare rendered by processing of photographic film. The server 400 mayproduce the image 700 and distribute it to be rendered at any suitablelocation. For instance, in the retail example, the server 400 maycommunicate the image 700 to a rendering device in the retail store sothat it can be seen by a manager. It should be noted however, thatimages need not always be rendered. For example, as discussed below, theSoftCamera may contain modules such as a messaging module that mayemploy the data contained in an image without rendering the image.

Like conventional light cameras, a lens of the SoftCamera has its own“shutter”. A lens is operational for a certain fixed amount of time, aninterval, called the shutter speed interval. The length of the intervalmay be calculated by taking into account the amount of computationalprocessing and communication processing that is needed to process theinformation collected by all the lenses in a DGA. Adding or removingcomputational resources varies the interval. A system clock is defined(for a DGA) that is called the Shutter Speed Clock (SSC) and this clockticks at the rate of the shutter speed interval for said DGA. At thestart of the interval the lenses in a DGA become operational, i.e.,information from the mobile devices within the DGA starts to bereceived. When the interval ends, the lenses become non-operational andthe receipt of information is terminated. Then a new interval starts andthe lenses become operational again.

Note that a SSC is defined for each DGA, which in turn may have severalmobile devices (and hence several lenses since the number of lensescorrespond to the number of mobile devices). However, we assume that theshutter speed for all lenses in a single DGA is the same. Thus, we haveone SSC for each DGA. The SoftCamera produces one image per click of theSSC for a given DGA.

Conventional light cameras use photographic film on which images arerendered. In the SoftCamera the notion analogous to photographic film isthe planogram. A SoftCamera renders the information it receives from alens (after processing said information) on the planogram provided to itas input.

The body of the SoftCamera is implemented as specialized software logicrunning on a server complex that is in network connection with mobiledevices in several DGA via any combination of wireless and wiredcommunication networks including, for instance, a private data networksuch as the Internet.

FIG. 10 shows an overview of an example of the SoftCamera from a generalperspective. The figure shows two DGAs “A” and “B” containing mobiledevices 100 and 200 respectively. As mentioned above, components withinsaid mobile devices acting as lenses transmit information 101 and 201respectively. The transmitted information reaches the camera body 500via aperture 800 said aperture controlled by Shutter Speed Clocks 600and 700 respectively. The camera body 500 has access to datasets 900such as a User List, CRM Data, Purchase Data and Trends (by way ofexample). The camera body 500 processes all the incident informationand, using the Planogram 950 and datasets 900 as specially pre-processedfilm, produces enhanced images such as 1001, 1002, etc. for DGA “A” andimages 2001, 2002, etc., for DGA “B”.

FIG. 11 shows more details of the internal modules of one example of theSoftCamera system (100). Image Formulator 200 is a module that takes acollection of datasets (e.g., CRM, Purchase, User list, Trends) andinformation produced by other modules to create images. Aperture Controlmodule 300 is controlled by the Shutter Speed Clocks (SSCs) module 800and opens and closes the shutter of a lens. SSC uses a system parameterto define the time interval constituting the shutter speed interval.This interval may be set by system policy, system administrator orinferred by the system based on the resources and demands being made tothe system. Thus the Aperture Control module accepts input signals frommobile devices in a DGA for one shutter speed interval. It sendsreceived information to the Messaging Module (MM) 500 for processing andcreating an image. It then resets the aperture and proceeds to acceptinformation from said mobile device in the succeeding shutter speedinterval.

The Messaging Module (MM), receiving information obtained from an image,formulates a collection of promotional offers that are to be sent tovarious mobile devices. It optionally uses the Recommender System 600for making hyper-relevant offers and for inferring items that a user maylike based on the user's linger-time and past purchase behavior.

The Analytics Module (AM) 400 operates on a collection of images ratherthan a single image. Its purpose is to analyze aggregate behavior overseveral images and derive “macro” analytics such as a determination ofthose areas in the DGA that see most traffic (“hot areas”).

The process for enhancing a planogram to serve as a “photographic film”for the SoftCamera involves the following steps.

I. The input is the planogram for a given DGA.

II. Add trend analysis of online feeds to the planogram.

III. Add the provided list of consumers to the planogram.

IV. Add CRM data to the planogram.

V. Add historical purchase data to the planogram.

VI. Add inventory data to the planogram.

For producing (initial) images the SoftCamera proceeds as follows. Wetake as input the planogram of said DGA.

Process Number: 1000

I. Start Shutter Speed Clocks, one for each DGA.

II. For each mobile device in the DGA do the following steps.

III. At the start of a first shutter speed interval, start receivinginformation from the lenses in the DGA. (The shutter is opened.)

IV. Collect information while the shutter is open.

V. Stop collecting information when the shutter closes.

VI. Send the collected information to be processed and the image to beproduced (explained below).

VII. Repeat the above steps ad infinitum.

As explained above, an image is an enhancement of a dataset. Such anenhancement may involve correlating the new information with theinformation previously contained in the dataset and appending newinformation to the dataset. In terms of producing an image that could berendered on a display device for human consumption, use can be made ofdata visualization technologies known in prior art.

For producing further processed images involves the following steps. Itis to be noted that the process described below sends its output to theMessaging Module that produces the final images.

Process Number: 2000

I. The input is the image produced by Process 1000 above and the list ofall users of the system.

II. Locate the BLE device in the input image.

III. Locate all the mobile devices in the input image. Find identifiersof users of said mobile devices (“user identifier list”).

IV. Find users that are common between the input list of all users andthe user identifier list of step III. Call this the list of “consumersin DGA”.

V. Compute the list of items proximate to every mobile device (“items ofinterest”).

VI. Compute “linger time” (explained below) for each user.

VII. Send list “consumers in DGA” to the “Messaging Module”.

VIII. Send list “items of interest” for each mobile device to the“Messaging Module”.

Messaging Module

The term “linger time” refers to an interval of time (in units of theshutter speed interval) that a mobile device spends in close proximityto one or more items in a DGA. The distance that constitutes “proximity”and the amount of time to be used as a parameter denoting “linger time”may be pre-determined or specified as configuration parameters by thesystem administrator.

For example, we may assume that the “linger time” is to be taken as 5shutter speed intervals and the proximity distance is to be taken as 5ft. from the center of a BLE device that is closest to a mobile device.

To compute the linger time for a given mobile device and a given BLEdevice we need to ascertain that said mobile device was within 5 ft. ofthe BLE device for 5 successive shutter speed intervals, i.e., in 5successive images received by the Messaging Module from the ProcessNumber 2000 described above. The “items of interest” list is then thelist of items designated by the planogram as being proximate to the BLE.(We are assuming that the original planogram gives locations of itemsand BLE devices and their spatial correspondence.)

The term “user identifier” refers to one or more identifiers thatuniquely describe credentials of a user that can be used to send amessage to the user, or that can be used to derive or infer credentialsof a user. As an example of a credential consider an email address suchas john@gmail.com or a Facebook or Twitter credential such asjohn@facebook.com or @john, etc.

The Messaging Module (MM) takes as input an image containing a list ofuser identifiers and their “items of interest” list in a given DGA.

MM performs the following main functions.

For each user in the user identifiers list it creates one or morepromotional “offers” (e.g., coupons, discounts, etc.), advertisements,recommendations or the like based on human curated recommendations. Itthen “sends” the offer or the like to a human curator for approval.Approved offers are sent to the indicated users. Alternatively, norecourse to a human curator may be undertaken and a completelyautonomous and automatic process may be followed.

In a more elaborate arrangement, the Messaging Module may have access toa Recommender System based on Machine-Learning technology that rankorders the “items of interest” list for a given user and recommends asubset of the list as being of “more interest” to said user.

Furthermore, said Recommender System may also recommend items that arenot on the “items of interest” list as possible items that said user isexpected to like, based on the past purchase data of said user. Recallthat in some cases past purchase data for users as explained above isassumed to be available to the original planogram as an input to thesystem.

In some embodiments information obtained from images in different DGAsmay be used by the messaging module (or other modules) for variouspurposes such as creating promotional offers or the like. For instance,inferences concerning a user's intent and interests may be determined byexamining the user's behavior when he first goes to retail establishmentA and then a bit later that day goes to retail establishment B. As asimple example, a user may first go to a pet store and then to abookstore. In this example, once in the bookstore an offer may be sentto the user concerning books about pets.

It is to be noted that prior art is replete with teachings ofrecommender systems based on machine-learning techniques such asCollaborative Filtering, etc.

Orchestration Script

Many buildings do not have access to the Internet and mobile deviceswithin such a building cannot communicate with Internet servers or cloudcomputing infrastructures while within the confines of the building.

The above real-world situation implies that mobile devices within a DGAmay not be able to act as lenses of the SoftCamera as described in theabove procedures. The above connection-free situation is analogous tolow-light situations faced by conventional cameras.

Conventional cameras overcome this situation by using flash lightmodules attached to a camera body.

In order to circumvent this problem in the SoftCamera we describe aprocess called the “Orchestration Script” (OScript). The general goal ofthe OS is to allow a connection-free operation by pre-loading a table ofconditions and actions into every mobile device that is within a givenDGA.

OScript is based on a table, called the Script Table (ST), each row ofwhich is associated with a BLE in a given DGA. For each BLE, the tablelists the items that are proximate to said BLE according to theplanogram of the DGA, the offers associated with said BLE pre-determinedby the store manager, and the Linger-Time offers associated with saidBLE (also pre-determined by store management). The ST may also containspecialized actions (“procedures”) such as “Get Cashier”, etc. A sampleOScript table is shown in FIG. 12. It is assumed that a pre-configuredprocess creates the Script Table (ST) for a given DGA, possibly curatedwith human help.

FIG. 12 shows a BLE device numbered #123 with proximate items 13 and145, along with offers associated with said device and specializedprocedures “Sales Help”, etc.

The OScript process has the following steps for a given DGA and itsassociated Script Table, ST.

Process: OScript

I. The process is initiated when a mobile device enters a geo-fencesurrounding a DGA.

II. Said mobile device is “marked”, say “D”.

III. The ST corresponding to device “D” is downloaded to the markeddevice.

IV. The device “D” executes the following procedure until it exits thegeo-fence.

-   -   a. Find the proximate BLE device from ST.    -   b. Execute the offers listed in column “Offers”.    -   c. Determine “linger-time”; if positive execute offer in column        “Linger-Time Offer”.    -   d. Determine if “Linger-Time” is greater than 5 minutes; if        positive execute “sales help”.

V. Send record of all executed actions and sightings of BLE to theSoftCamera body if the geo-fence has been exited and connectivity hasbeen restored.

The procedures referred to above such as “execute sales help” refer tosoftware programs (sometimes also called subroutines) that may specifyactions, e.g., send email messages to a list of individuals.

Those skilled in the art will recognize that the process OScriptdescribed above is an example of a state machine and that state machinesare a venerable part of computer science. Prior art teaches how toprogram and construct state machines to perform actions and executespecialized procedures based on complex pre-conditions. It is to benoted that the above example has been kept deliberatively simple fordidactical reasons.

Analytics Module

The Analytics Module (AM) is special software logic that takes as inputset of (i.e., one or more) images to derive analytics associated with aDGA. For example, we may be interested in those locations where mobiledevices spend more than a given amount of time, the most frequentlyvisited aisles, etc. Such traffic-related parameters are sometimesreferred to as “hot spots” that may be derived by analyzing a group ofsuccessive images, i.e., over a certain number of shutter speedintervals.

In terms of the overall flow of the SoftCamera process it may be statedthat the output of Process Number 2000 above serves as input to theAnalytics Module (AM) that in turn takes as input a collection of imagesand produces an enhanced image with the hot spot indications (by way ofone example of analytics).

Post-Visit Orchestration Script

Once a mobile device has exited the geo-fence marking the boundary ofthe DGA, special software logic is executed in the server complex.Recall that all images generated during the time interval that saiddevice was within the DGA are recorded (i.e., generated and saved by theSoftCamera) and are available for analysis.

The Post-Visit Orchestration (PVO) script is analytical software that istriggered when a mobile device exists a given DGA. Upon exit, the PVOscript when triggered analyzes any offers that were made to the mobiledevice whilst it was in the DGA, if any actions were taken by saidmobile device, offers made or accepted, time spent in the DGA, anypurchases made, linger time, items of interest, etc. Based on all theseparameters the PVO script determines if any additional offers are to bemade and presents its conclusions to a system administrator forapproval.

In the post-visit phase the SoftCamera images for a DGA may be providedto third-party providers through a suitable API and/or rendered onsuitable display devices for human consumption.

In the Orchestration Script phase (or the phase in which the mobiledevice is within the DGA) the images being generated for the DGA may berendered on suitable display devices wherein human operators mayassimilate the information being displayed and acted thereon.

A preferred embodiment of the claimed invention has been implemented ina cloud-based computing system with FIGS. 1-5 representing screen shotsof some of the aspects of the invention.

Illustrative Computing Environment

Aspects of the subject matter described herein may be described in thegeneral context of computer-executable instructions, such as programmodules or components, being executed by a computer. Generally, programmodules or components include routines, programs, objects, datastructures, and so forth, which perform particular tasks or implementparticular abstract data types. Aspects of the subject matter describedherein may also be practiced in distributed computing environments wheretasks are performed by remote processing devices that are linked througha communications network. In a distributed computing environment,program modules may be located in both local and remote computer storagemedia including memory storage devices.

FIG. 10 illustrates various components of an illustrativecomputing-based device 400 which may be implemented as any form of acomputing and/or electronic device, and in which embodiments of variousaspects of the SoftCamera as described above may be implemented.

The computing-based device 400 comprises one or more inputs 406 whichare of any suitable type for receiving media content, Internet Protocol(IP) input, activity tags, activity state information, resources orother input. The device also comprises communication interface 407 toenable the device to communicate with one or more other entity using anysuitable communications medium.

Computing-based device 400 also comprises one or more processors 401that may be microprocessors, controllers or any other suitable type ofprocessors for processing computing executable instructions to controlthe operation of the device in order to provide a search augmentationsystem. Platform software comprising an operating system 404 or anyother suitable platform software may be provided at the computing-baseddevice to enable application software 403 to be executed on the device.

The computer executable instructions may be provided using anynon-transitory computer-readable media, such as memory 402. The memoryis of any suitable type such as random access memory (RAM), a diskstorage device of any type such as a magnetic or optical storage device,a hard disk drive, or a CD, DVD or other disc drive. Flash memory, EPROMor EEPROM may also be used.

An output is also provided such as an audio and/or video output to adisplay system integral with or in communication with thecomputing-based device. A display interface 405 is provided to control adisplay device to be used in conjunction with the computing device. Thedisplay system may provide a graphical user interface, or other userinterface of any suitable type.

What is claimed is:
 1. A method for generating successive enhancedversions of a dataset, comprising: (i) during a first time intervaldetermined by a system clock, receiving over a wireless communicationnetwork from at least one mobile device one or more first signals fromwhich one or more locations of the at least one mobile device within ageographic area is derivable; (ii) generating a first enhanced datasetby incorporating the one or more locations of the one or more mobiledevices into a dataset that includes a representation of the geographicarea; and (iii) repeating steps (i) and (ii) for successive timeintervals to generate successive enhanced datasets at a rate determinedby the system clock.
 2. The method of claim 1 wherein the at least onemobile device comprises a plurality of multiple devices.
 3. The methodof claim 1 wherein the dataset comprises a planogram.
 4. The method ofclaim 1 wherein the first and successive enhanced datasets define eachdefine an image that is renderable on a display device.
 5. The method ofclaim 1 wherein the geographic area includes one or morelocation-deriving signal generating devices that wirelessly transmitlocation-deriving signals to the at least one mobile device, the one ormore first signals being based on information included in thelocation-deriving signals.
 6. The method of claim 1 further comprisingincorporating additional data into the first enhanced dataset.
 7. Themethod of claim 6 wherein the additional data includes data obtainedfrom one or more additional datasets.
 8. The method of claim 7 whereinthe geographic area includes a retail store and at least one of theadditional datasets includes customer relationship management data. 9.The method of claim 7 wherein the geographic area includes a retailstore and at least one of the additional datasets includes purchasehistory data.
 10. The method of claim 7 wherein the geographic areaincludes a retail store and at least one of the additional datasetsincludes trend analysis data.
 11. The method of claim 1 furthercomprising formulating a promotional offer based on the first and/orsuccessive enhanced datasets and sending the promotional offer to the atleast one mobile device.
 12. The method of claim 1 wherein the at leastone mobile device comprises a plurality of mobile devices and furthercomprising formulating a different promotional offer tailored for eachone of the mobile devices based on the first and/or successive enhanceddatasets and sending the promotional offers to the respective mobiledevices.
 13. The method of claim 1 wherein step (iii) comprises: (iv)during a second time interval determined by the system clock, the secondtime interval being subsequent to the first time interval, receivingover the wireless communication network from the mobile device one ormore additional first signals from which one or more updated locationsof the mobile device within the geographic area is derivable; (ii)generating a second enhanced dataset by incorporating the one or moreupdated locations of the mobile device into the dataset that includesthe representation of the geographic area.
 14. A method of forming animage, comprising: receiving over a wireless communication network froma mobile device a second signal indicative of a location of the mobiledevice within an environment represented by a planogram, the secondsignal being derived at least in part from a first signal generated by alocation-deriving signal generating device located in the environmentand detected by the mobile device; and generating an image that includesthe location mobile device overlaid on the planogram.